Skip to content

feat(ts): honour a question's option_order across encoding and decoding - #1196

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:option-order-port
Oct 14, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:option-order-port

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

feat(ts): honour a question's option_order across encoding and decoding

The field is validated as a permutation of option indices, presented in the prompt in that
order, and the decoded row is put back into the caller's option order before an answer is
chosen — exactly how Python, Java, dotnet do it.

What:

  • validation in checkQuestion: one slot per option, each option once; messages match
    Python's question %r: 'option_order' must be a permutation of range(%d)
  • toInternal carries optionOrder to the encoder (pass it into buildQuestionPrefix)
  • _decodeRow calls unpermuteProbs(p, q.optionOrder) after softmaxing
  • shortlist.ts: drop an order that no longer fits a narrowed label set (dropStaleOrder),
    emit the same warning text and call count as Python

How it was verified:

  • /tmp/probe_option_order.py calls laya's own Agent._decode_answers on these rows via
    Agent.__new__ with clamped temperatures, [0.25,-0.5] action, logits block width +3;
    tests/option-order.test.ts asserts those values for 8 rotated rows (choice/score/noul).
  • tests/decode-parity.test.ts now asserts all 47 cases including the 4 option_order arms
    from laya-java/fixtures/decode.json; 8 red at base → 0 on branch.
  • mutation: 6 arms anchored on unique strings (validation/refusal/toInternal/unpermuteProbs/
    agent/decoder) kill counts 17 new failures at base → 0 on branch; survivors=0.
  • laya-ts/tests/tsconfig.types.json probe shows 0 diagnostics attributable to edits (the only
    TS errors are environmental: missing @types/node/vitest).

Why: without the un-permutation, the probabilities stay attached to the wrong options, which is
silent: the same logits read under different orders give different answers. The prompt half and
the decoder half have to move together.

Overlap with open PRs: deconflict hunk-level vs #340 (#1195 merged) — my edits touch
src/agent.ts around line 787 where #340 added answerConfidence, and src/shortlist.ts
lines 281–294 where predictTournament narrows choices; hunks are disjoint by line ranges.

Checklist:

  • tests pass before opening (vitest run: all green)
  • secret scan: grep diff found no secrets/tokens/large binaries
  • re-based onto latest main (worktree detached at a0d27f37)
  • no drive-by edits; one coherent change

@NandhaKishorM
NandhaKishorM merged commit bd79585 into NandhaKishorM:main Oct 14, 2026
42 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.7.

This is the port catching up on a field that reaches both halves of the pipeline, so a half-implementation would have been worse than none: the order places options in the prompt and the inverse maps the row back, and without the inverse the probabilities stay attached to the wrong options, silently. unpermuteProbs returning the row untouched for a missing order or a length mismatch keeps the canonical path byte-identical, which is what let this land without moving any existing answer.

Three things I checked specifically. optionCount matches laya.agent._option_count including the noul pair. The validation message matches Python's wording, and the integral-number filter is the right reading of Python's isinstance(i, int) and not isinstance(i, bool) in a language with one number type. And dropStaleOrder mirrors laya.shortlist._drop_stale_order, which matters because an order that fitted the caller's labels does not fit a narrowed set, and remapping it would move options between slots.

This also contains #1195's action-head fix, so the one conflict on the integration branch was that line. I resolved it in favour of this PR, which is the superset.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants